Multivariate Clustering by Dynamics

We present a Bayesian clustering algorithm for multivariate time series. A clustering is regarded as a probabilistic model in which the unknown auto-correlation structure of a time series is approximated by a first order Markov Chain and the overall joint distribution of the variables is simplified by conditional independence assumptions. The algorithm searches for the most probable set of clusters given the data using a entropy-based heuristic search method. The algorithm is evaluated on a set of multivariate time series of propositions produced by the perceptual system of a mobile robot. Introduction Suppose one has a set of time series generated by one or more unknown processes, and the processes have characteristic dynamics. Clustering by dynamics is the problem of grouping time series into clusters so that the elements of each cluster have similar dynamics. Suppose a batch contains a time series of stride length for every episode in which a person moves on foot from one place to another. Clustering by dynamics might find clusters corresponding to “ambling,” “striding,” “running,” and “pushing a shopping cart,” because the dynamics of stride length are different in these processes. Similarly, cardiac pathologies can be characterized by the patterns of sistolic and diastolic phases; economic states such as recession can be characterized by the dynamics of economic indicators; syntactic categories can be categorized by the dynamics of word transitions; sensory inputs of a mobile robot can be merged to form prototypical representations of the robot’s experiences. The task of clustering time series can be regarded as the process of finding the partition, i.e. the set of clusters, best fitting the data according to some criteria. Typically, this task involves two steps: (1) model each time series to capture its essential dynamical features; (2) partition the set of time series by clustering. Our approach uses one of the simplest representations of a time series: a first order Markov chain (MC). A MC assumes that the probability distribution of a variable at time is independent of the variable values observed prior to time (Ross 1996). Furthermore, we regard the task of finding the best partition of the data Copyright c 2000, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. as a statistical model selection process. Smyth (1999) applied this idea to clustering time series and Sebastiani et al. (1999) devised a Bayesian model-based algorithm to cluster higher-order MCs. The algorithm, called Bayesian Clustering by Dynamics (BCD), has been successfully applied to cognitive robotics (Sebastiani, Ramoni, & Cohen 2000; Cohen et al. 2000), simulated war games (Sebastiani et al. 1999), behavior of stock exchange indices, and unsupervised generation of musical compositions. These applications suggest that even a very simple MC representation of a dynamic process is powerful enough to capture common aspects of different time series. Furthermore, an appealing feature of BCD is an entropy-based heuristic that makes the search over the space of partitions very efficient. In its current formulation, BCD is limited to the univariate case, that is, the algorithm is able to cluster the behaviors of only one variable at a time. But what if the problem at hand is multivariate, that is, it is represented by simultaneous time series of several interacting variables? The assessment of a battlefield situation is done on the basis of several, possibly interacting, factors, like force ratio, number of engaged units, total forces mass, and so on. Similarly, a sensory experience of a mobile robot is given by the simultaneous values of several sensors, and the experience itself can be identified by the correlation among subsets of these variables. Suppose one wants to cluster a set of multivariate time series of discrete variables, each taking values. The straightforward solution is to convert the problem into a univariate one by defining a single variable taking as values all combinations of values of the variables and then applying the univariate case algorithm (Sebastiani et al. 1999). Unfortunately, this solution is hardly scalable because the number of states of this variable grows exponentially with the number of the original variables. The solution we present in this paper is a novel clustering technique for multivariate time series, called Multivariate Bayesian Clustering by Dynamics (MBCD). The clustering algorithm is model-based, as it represents a clustering as a probabilistic model, it is Bayesian, as both the decision of whether grouping MCs and the stopping criterion are based on the clustering posterior probability, and it uses an entropy-based heuristic to reduce the search space over a subset of possible partitions. The clusters of dynamics produced by the algorithm are sets of MCs, which are assumed to be conditional independent given From: AAAI-00 Proceedings. Copyright © 2000, AAAI (www.aaai.org). All rights reserved. cluster membership. Thus, they capture dynamics involving simultaneously all the variables but the conditional independence assumption makes the algorithm scalable to large data sets. The algorithm is tested on multivariate time series of propositions produced by a mobile robot perceptual system and produces clusters which are significantly different from random clustering and in agreement with human clustering. Theory Suppose we have a set of multivariate time series. Each multivariate time series is a set of univariate time series recording values of variables . The multivariate clustering algorithm can be outlined as follows. Given the univariate time series, construct a MC for each series and replace each of the multivariate time series by a set of MCs. Rank the sets of MCs in decreasing order of distance, merge similar sets of MCs into clusters if the merging increases a scoring metric, and repeat the procedure until a stopping criterion is met. The first step is the estimation of a MC from a univariate time series and it is considered next. Markov Chains Suppose that, for a variable , we observe the time series "! $# $% & $% ' , where each (% is one of the states ) * of . The process generating the series is a (first order) MC if the conditional probability that the variable visits state + at time , , given the sequence "! $# "-.& ' , is only a function of the state visited at time ,0/1) . Hence, we write 2 3-